…hrough OpenEnv

Trains against a Harbor task through mini-swe-agent running in an E2B sandbox. The agent owns its own
loop; TRL stands up an endpoint, lets it drive, and reads back the captured token ids and logprobs. That
is what makes an installed harness trainable without reimplementing it, and it is the difference from
examples/grpo_harbor, which runs Harbor tasks against harnesses written inside TRL with TRL owning the
loop.

mini-swe-agent is the default on measured grounds rather than taste: across a 15-harness sweep on the
same 50 tasks it was the most accurate and the most turn-efficient, its prompt re-render is byte-exact
against the engine's prompt_token_ids, and it is the only harness that can express a step limit. The
re-render matters because TRL rebuilds each prompt locally, and for three of twelve harnesses measured
that drifts (claude-code +2 tokens, gemini-cli +2, kimi-cli -10 per tool call) -- invisible for eval,
forking the trajectory every turn when training.

The step limit is not a cost control. Every turn re-sends the whole conversation, so a rollout's packed
length grows with the SQUARE of its turn count; unbounded 58-turn rollouts were enough to OOM the loss
step on an 80 GiB card.

The docstring states one caveat rather than hiding it: on this path HarnessRolloutOutcome carries a
single verifier scalar, not the component dict, so a 'submission' term giving partial credit is
unavailable and the reward is all-or-nothing. On a suite the model solves ~16% of the time that means
most groups score identically and those steps teach nothing, so the example says to shape component
rewards where the suite emits them and otherwise to pick tasks the model solves sometimes.

Brings the whole stack up inside one container: openenv harbor serve on CPU, vllm serve on GPU 0, the
trainer on GPU 1. Self-contained rather than importing its sibling, because hf jobs uv run uploads a
single script.

Everything lives in one job because AsyncGRPO syncs weights into vLLM over NCCL, which needs both on the
same host's GPUs. That leaves only the OpenEnv server placeable, and keeping it here puts the proxy's hop
to vLLM on localhost; hosting it on a Space via `openenv harbor push` works but adds a public hop to
every model call.

The tunnel is not a workaround for missing infrastructure. Jobs can publish a port at
<job_id>--<port>.hf.jobs, but access needs an HF token, and the agent's Authorization header already
carries its rollout session key -- that key IS how the proxy routes concurrent rollouts, so it cannot
carry a second credential. An unauthenticated outbound tunnel needs no ingress at all.

The readiness check verifies the tunnel SERVES THE PROXY, not that a port answers. A tunnel whose
forwarding process dies keeps resolving and returns the provider's error page; agents then get HTML where
an OpenAI endpoint should be, make zero model calls, and every rollout comes back unscorable while the
server's own /health still reports healthy. The URL is only accepted after a request through it returns
our health document. The parser was replayed over 32 real server logs and matches all 12 published URLs,
gradio and cloudflare alike.

The mounted bucket holds checkpoints and HF_HOME, and says so loudly when absent rather than losing them
silently. Deliberately not the sandbox templates: those are keyed by image hash on the provider's side
and already persist across jobs, which is the expensive warm step.

Restructured to match the pattern the opencode recipe already uses for Hugging Face Jobs (described in
sergiopaniego's TRL x OpenEnv x Harbor post, with the launcher published as a gist): a small launcher
wraps the processes a Job cannot start on its own, and the training script is DOWNLOADED rather than
duplicated.

That last part is the reason for the change. The previous commit added a second 489-line file that
restated the whole training path, so the two could drift and a reader had to diff them to see what was
actually different. Now async_grpo_harbor.py is the single canonical script, byte-identical whether it
runs locally or in a Job, and launcher.py holds only what is genuinely Jobs-specific.

Ours needs three processes where the opencode launcher needs two, because the Harbor dataset and the
capture proxy are served by openenv harbor serve. It also tunnels a different hop: opencode tunnels
vLLM, since its in-sandbox proxy calls the engine directly, whereas here the sandboxed agent calls the
capture proxy and the engine stays entirely private on localhost.

--train-script-url exists because the equivalent opencode launcher still points at
examples/scripts/openenv/opencode_hf_sandbox.py, a pre-reorg path that now 404s. A stale URL surfaces as
a download failure minutes into a paid job, so this one is overridable and can be pointed at a branch
before the example is merged.

Carried over from the audit of the deleted file: vllm is declared, the sandbox template is warmed by one
serial rollout before any group runs concurrently, there is a GPU-count guard, the tunnel readiness check
verifies a request THROUGH the tunnel reaches the proxy, and the run name never falls back to a constant.

…prompt

In the loop-owning path the agent runs its own tool loop in a sandbox and TRL
reads back what it did. Rebuilding each turn's prompt with apply_chat_template to
recover the token ids produces a DIFFERENT string from the one the engine scored.
Measured on Qwen3.5-4B, the rebuilt prompt matched the engine on 0 of 28 turns.

Nothing errors when that happens, which is the problem. Turns that should chain by
exact token prefix instead look like unrelated short rollouts, so one conversation
fragments into several and every fragment still trains. Two production runs spent
a night each in exactly that state with healthy-looking reward curves; the tell was
samples_per_rollout equalling turns_mean exactly, i.e. 100% forking.

_turns_from_trace now takes `prompt_token_ids` from the trace entry and raises when
it is absent, naming the two vLLM flags that produce it. A hard failure rather than
a fallback on purpose: the fallback IS the bug, and it is invisible. An engine
without --return-tokens-as-token-ids yields rollouts that look completely normal
and carry nothing to train on, so the run has to stop at the first rollout instead
of discovering it at the first weight update. The tokenizer argument is gone from
that path -- there is nothing left for it to do.

TurnRecord gains `output_mask`, and _SampleBuilder.append_turn honours it instead
of hardcoding 1 across the completion. This is not cosmetic: capture masks a turn
OUT when its logprobs were rejected on ingest, while keeping its tokens as context
and zero-filling the logprobs. Without the mask those positions train against
logprob 0.0, i.e. p=1.0 -- a confident target for a turn we explicitly could not
trust.

Requires the matching OpenEnv change, which puts `prompt_token_ids` and `loss_mask`
on TraceEntry. Against an older OpenEnv this raises immediately and says which
flags are missing, which is the intended behaviour rather than a regression.

Note for anyone tempted to prototype this as a monkeypatch: it cannot work. The
rollout loop runs in a multiprocessing child created with `spawn`, which re-imports
every module from scratch, so a parent-side rebind of a module function is simply
lost. A patch that mutates the tokenizer OBJECT survives, because that is pickled
into the child; one that rebinds a function does not. That asymmetry is why the
re-rendering went unnoticed for so long -- the patch logged its install line and
its counters never moved.

# Conflicts:
#	docs/source/example_overview.md

CI's code-quality job is `pre-commit run --all-files`, and its `doc-builder-style` hook rewraps
docstrings to max_len 119. My `_turns_from_trace` docstring wrapped at a narrower width, so the hook
modified a file and the job failed.

Worth recording for next time: running ruff locally does NOT predict this job. ruff passed on all
five of my changed files while the real failure was doc-builder rewrapping one line. The gate is
pre-commit, so reproduce it with pre-commit (or at least the pinned doc-builder rev) rather than with
ruff alone. The notebook errors ruff reported locally were a red herring -- those files are
byte-identical to upstream and I touched no notebook.